So far the networking of virtual machines has usually been managed by configuring a Virtual Ethernet Bridge (VEB) in the host, i.e. the good old implementation of 802.1D in the Linux Kernel:

A Virtual Ethernet Bridge (VEB) is a capability within a physical end station that supports local bridging between multiple virtual end stations and (optionally) the external bridging environment.

While this works well and in some cases is the best solution, in other setups performances or policy needs dictate that the traffic between virtual machines is forwarded to an external bridge (the Ethernet switch connected to the host) and back without having in the host a real bridge learning MAC addresses from its ports and implementing the STP. Enters the VEPA:

A Virtual Ethernet Port Aggregator (VEPA) is a capability within a physical end station that collaborates with an adjacent, external bridge to provide bridging support between multiple virtual end stations and external networks. The VEPA collaborates by forwarding all station-originated frames to the adjacent bridge for frame processing and frame relay (including "hairpin" forwarding) and by steering and replicating frames received from the VEPA uplink to the appropriate destinations.

Support for VEPAs has been added to Linux 2.6.33 with a simple change to the macvlan driver. The next step will be to use the macvtap driver (available in 2.6.34), which exposes a tap character device usable by kvm-qemu and "connected" to a macvlan-like interface.

If you have a recent Dell Laptop (I tested this on a Latitude E6400) and even if VT support is enabled in the BIOS setup you get this error when you try to load kvm-intel:

kvm: disabled by bios

then make sure that the "Trusted Execution" feature in the "Virtualization Support" menu is disabled. Should I blame Dell or the KVM developers?

Recently some Mozilla developers (1, 2) have been trying to justify the lack of support of the HTML 5 <video> element in Firefox for the system-provided codecs (e.g. GStreamer, DirectShow and so on). While I fully agree that patented codecs like H.264 are very bad for the Internet, it is not clear to me how watching H.264 video with the proprietary Flash plugin would help promoting unencumbered codecs more than watching H.264 video without using non-free software. Beware: if your position is that there is no need to watch H.264 Youtube videos then please refrain from wasting your and my time with a reply, because you are missing the point (and probably many others as well).

• 802.1q (VLANs).
• 802.1ax (link aggregation) between the pair of switches in a chassis.
• 802.17 or another simple Ethernet ring protocol, but I can live with a decent 802.1d implementation (STP) as well.
• IGMP and MLD snooping, while not yet a mandatory requirement, would be appreciated.

Why link aggregation? It is the only simple solution for having complete fault tolerance without configuring the STP in the servers (which is annoying, because Linux does not implement portfast) or even worse OSPF. That's it. I do not need routing protocols or even L3 routing at all, just a simple switch. But apparently the vendors would rather sell me an high end device...

After one year of development, today I uploaded my first package to CPAN: version 4.0 of Bryar.pm, a modular and extensible blog engine powered (by default) by Template Toolkit and originally written by Simon Cozens. I am officially a Perl developer! This is the software which I use for my own blog. Among the many changes in this release: caching, support for FastCGI, improved support for HTTP validators and a sitemaps generator.

My employer uses the IBM Tivoli Storage Manager for backups and since we use Debian everywhere possible we needed packages of its client program, which is only distributed as RPMs. The initial conversion made with alien was unsatisfactory for many reasons, and the ugly original packaging did not help either. For this reason I spent a few hours to create an high quality Debian package for the Tivoli Storage Manager client. The client binaries cannot be redistributed, so when it is built from source it unpacks the RPMs and copies from there the files it needs to build the binary package. This work has been sponsored by Seeweb, a provider of web hosting, dedicated hosting and colocation services.

• Marco d Itri about adding support for yet-unknown-to-kernel-drivers devices. More specifically: how to make udev tell ftdi-sio that it should accept a certain VendorID/ProductID combination in addition to previously supported ones.
• Thomer M. Gil about ICMPTX, a combination made from proxy and server that tunnels IP through ICMP. Nice thing if you need a connection from some NATed and heavily firewalled location I guess. (Thanks go to Runa Sandvik for finding this.)
• Non-technical in a way, but still a nice link: Image of the Shuttle transiting the sun, thanks to Pablo Lorenzzoni for this link.
• Another post by Marco d Itri that explains how to install Debian on a SheevaPlug. Not that I need this info now, but it might come in handy some day.

That s it for now, will update the post if I find more interesting links in the next few days.

Some common classes of USB devices are supported by the same drivers because they share an identical or almost identical hardware design, but since each manufacturer uses its own Vendor and Product IDs the drivers need to be updated for each one. Some popular examples are 3G modems and devices which present a serial interface. As an interim solution until the kernel can be updated, the Linux device model provides a sysfs parameter called new_id which can be used to notify a driver that it should recognize a VID/PID pair as supported. The following example uses udev to add support for the Sheevaplug to ftdi_sio, the driver for the widely used USB to UART chips manufactured by FTDI. First we need a rule to automatically load the driver when the device is plugged in: udev cannot load it by itself since it does not know about the Sheevaplug VID and PID:

# if no driver has claimed the interface yet, load ftdi_sio
ACTION=="add", SUBSYSTEM=="usb", ENV DEVTYPE =="usb_interface", \
        ATTRS idVendor =="9e88", ATTRS idProduct =="9e8f", \
        DRIVER=="", \
        RUN+="/sbin/modprobe -b ftdi_sio"

And then we instruct udev to add the new VID and PID after the driver has been loaded:

# add the sheevaplug VID and PID to the list of devices supported by ftdi_sio
ACTION=="add", SUBSYSTEM=="drivers", \
        ENV DEVPATH =="/bus/usb-serial/drivers/ftdi_sio", \
        ATTR new_id ="9e88 9e8f"

While we are configuring udev we can also make it create a symlink for the console interface, since the FTDI chip in the Sheevaplug provides two serial interfaces: interface 0 is connected the JTAG lines and interface 1 to the serial console:

# create a convenience symlink for the console device
ACTION=="add", KERNEL=="ttyUSB*", \
        ATTRS interface =="SheevaPlug JTAGKey FT2232D B", \
        ATTRS bInterfaceNumber =="01", \
        SYMLINK+="sheevaplug"

This last rule shows an important misfeature of udev: while the keys ending in "S" can match nodes at any level of the devices chain, the matched nodes must be all at the same level. In this case this means that a single rule cannot have keys matching both the interface number and the VID and PID: this would require multiple rules and GOTO directives, so my rule matches the interface node which is available at the same level.

This document explains how to use deboostrap --foreign to build from scratch and install a minimal Debian system on the Sheevaplug (but it is equally applicable to many other embedded systems). The only piece missing to be able to install from bare metal is a working kernel and initramfs pair, so this document assumes that you already have a kernel which can mount the root file system from a USB key. For the Sheevaplug, the factory kernel in the internal NAND flash will work fine. The Debian kernel package puts us in a chicken and egg situation: it cannot mount the root without an initramfs (because all drivers are modular), but we cannot build one until we can boot the system. I suppose that, as an exercise of style, somebody could create a minimal initramfs with busybox and the kernel modules needed to mount the root and then mount it manually, but if you really lack a working kernel then it's probably faster to boot the file system image for the first time using qemu and the "versatile" debian-installer target. The first step is to run the first stage of debootstrap on another system (which does not need to be ARM), to build a temporary root file system. I use the --variant and --include options to build the smallest possible system with network support, but they can be omitted.

debootstrap \
         --foreign --arch=armel \
         --variant=minbase \
         --include=linux-image-2.6-kirkwood,module-init-tools,udev,netbase,
                   net-tools,ifupdown,iproute,whiptail,vim-tiny \
         sid ./target/ http://ftp.it.debian.org/debian/

As usual when using debootstrap, some manual configuration is needed:

cd target/
echo bokassa.mi.bofh.it > etc/hostname
vi etc/network/interfaces
vi etc/resolv.conf
echo LANG=C > etc/default/locale
echo '127.0.0.1 localhost' > etc/hosts
cat <<END > etc/fstab
/dev/root / ext3 noatime 0 1
tmpfs /tmp tmpfs defaults 0 0
#/tmp /var/tmp none bind 0 0
END

I took some hints from http://wiki.debian.org/ReadonlyRoot to reduce writes to the flash file system:

# the blkid cache is evil anyway
ln -s /dev/null etc/blkid.tab
# this link will be created anyway by the package
ln -s /var/lib/initscripts/nologin etc/nologin
# it's better to not use a static mtab with modern kernels
ln -s /proc/mounts etc/mtab
# unless you plan to use multiple removable network interfaces, persistent
# interface names are not needed
: > etc/udev/rules.d/75-persistent-net-generator.rules

Two small workarounds are needed:

# if this is not set then the preinst of linux-image-* will die trying
# to ask a debconf question in the debootstrap second stage
echo 'do_initrd = yes' > etc/kernel-img.conf
# a workaround for bug #520742
: > etc/udev/disabled

Now you can format a USB pen drive and copy the root file system on it:

mke2fs /dev/sdb1
tune2fs -i 0 -c 0 /dev/sdb1
mount /dev/sdb1 /mnt/
rsync -aH target/ /mnt/
umount /mnt/

After powering the plug press a key to get to the U-Boot prompt and use the factory kernel to boot from USB:

setenv bootargs console=ttyS0,115200 root=/dev/sda1 rootdelay=10 panic=10 init=/bin/bash
run bootcmd

Then you can run the second stage of debootstrap, finish the bare minimal configuration and start init:

./debootstrap/debootstrap --second-stage
# disable the gettys for /dev/tty[1-6], which do not exist
vi /etc/inittab
# and add one for the serial console
echo 'T0:2345:respawn:/sbin/getty -L ttyS0 115200 linux' >> /etc/inittab
# do not forget to set the root password or you will not be able to login...
vi /etc/shadow
rm /etc/udev/disabled
exec /bin/bash
mount / -o ro,remount
exec /sbin/init

Congratulations, now you can login on the system and start to install the packages you like:

echo 'deb http://ftp.it.debian.org/debian/ unstable main' > /etc/apt/sources.list
echo 'APT   Install-Recommends "false";  ;' > /etc/apt/apt.conf.d/no-recommends
apt-get update
# dash is much faster
apt-get install dash
dpkg-divert --add /bin/sh
ln -sf dash /bin/sh
# I do not like these, YMMV
rm /vmlinuz /initrd.img
rmdir /selinux /srv
# the kernel is able to read the time from the RTC by itself, and this
# will allow using a read only root
echo HWCLOCKACCESS=no >> /etc/default/rcS
# faster boot
echo CONCURRENCY=shell >> /etc/default/rcS
# useless on this platform
mv /etc/rcS.d/S01glibc.sh /etc/rcS.d/K01glibc.sh
dpkg-reconfigure tzdata
apt-get install openssh-server screen iptables wget less

U-Boot requires that the kernel and initramfs have a special header:

apt-get uboot-mkimage
cd /boot/
mkimage -A arm -O linux -T kernel  -C none -a 0x00008000 -e 0x00008000 \
  -n Linux-2.6.29-2 -d vmlinuz-2.6.29-2-kirkwood uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 \
  -n initramfs -d initrd.img-2.6.29-2-kirkwood uInitrd

Now you can reboot and configure u-boot to boot from the USB device. This is the variables scheme I like and it is not mandatory in any way, so I recommend that you read and understand the existing configuration before modifying it (some lines have been broken for readability):

setenv bootargs_nand $(bootargs)
setenv bootcmd_nand $(bootcmd)
setenv bootargs_root root=/dev/sda1
setenv bootargs_misc 'ro panic=10
  mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)'
setenv bootcmd_usb 'usb start; ext2load usb 0:1 0x00200000 /boot/uImage;
  ext2load usb 0:1 0x01100000 /boot/uInitrd'
setenv bootcmd 'setenv bootargs $(console) $(bootargs_root) $(bootargs_misc)
  $(bootargs_more); run bootcmd_usb; bootm 0x00200000 0x01100000'

Since now you are not using anymore the Marvell kernel tree it is mandatory to set these two variables and reboot:

setenv mainlineLinux yes
setenv arcNumber 2097
saveenv
reset

Installation to SD is identical, but requires an updated version of U-Boot with SD support. Due to a kernel bug which prevents autoloading the driver you will need to add mvsdio and mmc_block to /etc/initramfs-tools/modules and rebuild the initramfs. (Yes, the naming theme I use at home is "african dictators"...)

In a service provider environment it is often required by customers to support old server installations for much longer than it is recommended by vendors. Recently I had to move some Xen VPSes running Debian 3.1 to a modern Xen host running lenny and the 2.6.26 kernel. Since Xen is a FPOS and has been unable for years to maintain a stable ABI, I always use in the domUs the same kernel version of the dom0. The Debian Xen kernels require using an initramfs to load the paravirtualized block device driver and mount the root, at least. This would usually not be a big deal, but in this case breaks the VPS because the lenny version of udev is started in the initramfs and the system has either no udev or the old sarge version. The most immediately visible effect is that /dev/pts does not exist, which makes ssh less than useful. This is my solution:

dpkg --purge hotplug udev
printf '#!/bin/sh\nmkdir /dev/pts\n' > /etc/rcS.d/S02mkdir
chmod +x /etc/rcS.d/S02mkdir

My SE Linux Play Machine [1] has a file named thanks.txt for users to send messages to me [2]. On a number of occasions people have offered to give me things in exchange for the password for the bofh account (the one with sysadm_r privileges). I ve been offered stolen credit cards, a ponzi scheme of root access to servers on the net, and various other stuff. Today I received an amusing joke entry: Hello Kind Sir, I am Dr. Adamu Salaam, the the bank manager of bank of africa (BOA) Burkina Faso West I am sending you this message about the $3.14159 million dollars in bank account number 2718281828450945. I will give you this money in exchange for the password to the bofh account. The amount of money is based on the value of Pi. The account number is based on the mathematical constant e [3]. It s a pity that the author of that one didn t sign their real name. Whoever created that should have claimed credit for their work.

• [1] http://www.coker.com.au/selinux/play.html


• [2] http://doc.coker.com.au/computers/thankstxt-on-my-play-machine/


• [3] http://en.wikipedia.org/wiki/E_(mathematical_constant)

All releases of udev older than 141 allow a local user to instantly get root privileges (CVE-2009-1185). I am not aware of a publically available exploit. Workaround: kill udevd(8).

It s now been almost a month since I got my Kindle 2 and wrote about my first impressions of it. In that amount of time, I figure I have read, on the Kindle, the equivalent of 1500 to 2000 printed pages. I ll start with my personal reactions to the device, and then move on into some less subjective observations about it. What I ve Used It For To give you a quick idea of what I m reading:

• David Copperfield (downloaded at feedbooks.com)
• Part of Houston Smith s Why Religion Matters, purchased at the Amazon Kindle Store (currently reading it)
• Cory Doctorow s Eastern Standard Tribe (from feedbooks)
• Most of Simon Travaglia s funny BOFH series through 2000 (built myself)
• Various other bits and pieces of things here and there.

The Gut Reaction I want everything to be available on the Kindle, and I want to read everything on it. Whether that s just some excitement at a new device (still possible after a month, I guess), or something more lasting, I m still a little unclear about but I suspect it s something more lasting. I have read far, far more since I have had the Kindle than I had for quite some time prior, excepting a certain Hitchhiker s Guide to the Galaxy marathon one Christmas vacation. This leads me to think that the Kindle s main competitor isn t the Sony PRS-505, but rather the XBox, Wii, Netflix, and Hulu. Not that I watched that much TV to start with, but I haven t even been keeping up with the Daily Show since I got the Kindle. Not only that, but it prompted me to go to the local used bookstore (all proceeds go to charity) and buy a paper copy of David Copperfield for $10, so Terah could read it with me while I reread it on the Kindle. So it seems that Amazon s stats which show that people tend to read more once they get the Kindle, and don t cut down on paper book buying are about right in my case. Plus there s the fact that I ve loaded up the Kindle with over a hundred free books of all sorts of descriptions, some from Amazon s own free book catalog (extensive in itself), and others from Feedbooks, Gutenberg, or other places. That s taken up some time itself, and I now have achieved what I tend to achieve with all devices like this (iPod, MythTV, etc): more material than I can ever possibly get through. Why I Like It So, what exactly IS it about this thing that I like? Part of it has to be the size of the device. Granted, I wish the screen were larger, but even with the leather case, the device is a pretty nice size. I find that I keep two or three books that I m reading at the top of my list, and it s great to be able to carry around a single device and be able to select from a book as the mood strikes. Part of it is the built-in dictionary and search tools, which I ve already discussed. In the end, Amazon s execution of the concept is just done really, really well. It doesn t require some sort of proprietary Windows app like Sony s reader does. It doesn t even require a PC at all. It acts as reliable as a book, and that s something. The highlight/clippings feature is also pretty nice. You can highlight a section of text, which then causes it to be rendered with an underline on-screen just like you can underline something in a book. Better yet, when you re reading the book, you can call up a list of all such passages (as well as your bookmarks, etc), see them with an excerpt of text, and jump directly to them. No more need for millions of tiny pieces of paper marking a page that contains a marking. Even better than that, the Kindle creates a .txt (yes, plain text) file in the documents/ directory for you. Each time you bookmark or highlight a passage, it adds a little summary of the passage to that file, as well as information about the location of it. For highlights, the summary is the content of the highlight itself. So if you re going to go blog about the book later, there are a bunch of ready-made quotes you can cut-and-paste later. You can read that file both on the Kindle and on a PC. Sweet. So, I guess I would say that the Kindle makes it easier to read complex texts, and fun to read lighter ones. Availability of Content There are several ways you can get content onto the Kindle. The first, of course, is Amazon s own Kindle Store. Quite the clever bit of engineering there. You hit buy , and generally by the time you can walk over to the Kindle and pick it up, the book s on it, thanks to the integrated cell-network wireless modem. Subscriptions to newspapers and magazines work similarly each issue is automatically delivered in the middle of the night. However, it seems that the magazines such as New Yorker and Atlantic aren t exactly complete replicas of the print editions for some reason. The pricing of Kindle content is generally cheaper than paperback editions of books, but not by much. I think it should be more heavily discounted than it is, and part of that may be due to the large commission Amazon takes on things. Ironically, Amazon also has a tremendous array of free public domain and copyrighted books, and an equally staggering number of books available for under $1. For the classics, and other things out of copyright, my favorite place to turn is Feedbooks. They ve got ebooks available for download for just about any format, including Kindle, and generally do a good job of providing a real linked Table of Contents and the like. Kindle can read DRM-free content in .txt or .mobi format. I ve also built my own ebooks for the Kindle using Mobiperl. It takes an ePub source directory and makes a MobiPocket file out of it, which can be installed on the Kindle directly. For things I find on Project Gutenberg, GutenMark is a nice tool to create a good-looking HTML representation. I also use quote-fixer to help with my other pet peeve: straight quotes and lack of em-dashes on a device that can, and should, display both. Screen I m still of two minds about the screen. On the one hand, it doesn t give me eyestrain like reading on a PC, laptop, or cellphone screen does. Also, it looks remarkably book-like. A well-prepared Kindle etext feels so professional on the screen, so real, so lovingly-prepared, that it s fun to read. The Feedbooks David Copperfield is an excellent example of this, as is Houston Smith s Why Religion Matters. There are several things I don t like about the screen. One is that, while reading a book, it leaves a margin around the edge of the text. I can t figure out why it doesn t use all the available screen real-estate, especially since its built-in web browser does. Another is that, while not terribly reflective, it is still somewhat reflective, and glare from the sun or a light at a bad angle can make the screen unreadable unless you turn it a bit to a different angle. But my biggest gripe about the screen is that it s not as high-contrast as I d like. Still, I very much enjoy the Kindle, so I guess it s not as huge a problem as I think it could be sometimes. Pet Peeve I get really annoyed when I download content for the Kindle and discover that it doesn t have a linked Table of Contents (meaning I can click on a chapter and go directly to it). I m even more annoyed when it uses straight typewriter quotes instead of the angled typesetter s quotes or smart quotes . The Kindle is capable of beautifully rendering them, as well as things such as em dashes, and when I see what is obviously some cheap attempt at a conversion of some ancient ASCII document, rather than a proper attempt to make a good-looking book, I get annoyed. And so it was that I requested, and got, a $0.80 refund for my purchase of Sherlock Holmes: The Complete Collection and instead paid $3.60 for The Complete Sherlock Holmes. Yes, I could have spent a few hours converting all these from Gutenberg, but $3.60 is so cheap that it wasn t worth it. Customer Service Amazingly, Amazon s customer service for the Kindle is not their usual Asian form letter service. It s Americans, who seem to read, understand, and care about problems. I was pleasantly surprised. Too bad the rest of Amazon doesn t learn a thing or two from it. Conclusion I think the Kindle is worth it. It will pay for itself exceptionally quickly for people like me that don t live anywhere near a large library, and enjoy reading out-of-copyright material. Even if it doesn t pay for itself that quickly, the convenience of being able to carry several actively reading books with you on a single small device is pretty nice. No more pile of books in my suitcase for travel.

This is my recipe for easily running kvm as an unprivileged user without using sudo and custom network configuration scripts. /etc/network/interfaces:

iface kvmnet inet static
        address         10.5.0.1
        netmask         255.255.255.0
        broadcast       10.5.0.255
        pre-up          vde_tunctl -u md -t $IFACE
        post-down       vde_tunctl -d $IFACE
        up              echo 1 > /proc/sys/net/ipv4/ip_forward
        up              iptables --table nat --append POSTROUTING --jump MASQUERADE --source $IF_ADDRESS/$IF_NETMASK
        down            iptables --table nat --delete POSTROUTING --jump MASQUERADE --source $IF_ADDRESS/$IF_NETMASK

~/bin/kvm:

#!/bin/sh -e
iface='kvmnet'
macaddr='DE:AD:42:00:00:01'
model='virtio'
exec kvm \
    -net nic,vlan=0,macaddr=$macaddr,model=$model \
    -net tap,vlan=0,ifname=$iface,script=no,downscript=no \
    "$@"

Configuring a PV network interface is optional, but it is more efficient and if the guest uses udev and a modern kernel it will just work. Bonus tip: use "-vnc :0 -usbdevice tablet -serial telnet:127.0.0.1:4444,server,nowait" for a headless guest which if needed will still have proper consoles.

It s been quite the week. Stomach Flu Last Friday, my stomach was just starting to feel a little odd. I didn t think much off it a little food that didn t go over well or stress, I thought. Saturday I got out of bed and almost immediately felt like throwing up. Ugh. I probably caught some sort of stomach flu. I was nauseous all day and had some terrible diarrhea to boot. I spent parts of Saturday, Saturday night, Sunday, and Sunday night supervising some emergency downloads as the BOFH would say. By Sunday afternoon, I thought I was doing good enough to attend a practice of the Kansas Mennonite Men s Choir. I made it through but it wasn t quite as up to it as I thought. Monday morning I woke up and thought the worst was behind me, so I went to work. By evening, the worst clearly was not behind me. I was extremely cold, and then got very hot a few hours later. Tuesday I left work a little early because of not feeling well. Servers Wednesday a colleague called me at home before I left to say that the ERP database had a major hiccup. That s never good. The database is this creaky old dinosaur thing that has a habit of inventing novel ways to fail (favorite pastime: exceeding some arbitrary limit to the size of files that no OS has cared about for 5 years, then hanging without telling anybody why). My coworkers had been working on it since 5. I went into the office and did what I could to help out, though they had mostly taken care of it. Then we went to reboot the server. It didn t come back. I/O error on sda just after init started, and it hung. Puzzled, as it just used that disk to boot from. Try rebooting again. This time, I/O error as the fibre channel controller driver loads. Again, puzzled as it just used that controller to load grub. Power cycle this time. And now the server doesn t see the fibre channel link at all. Eep. Check our fiber optic cables, and power cycle again. And THIS time, the server doesn t power back up. Fans whir for about a second, then an ominous red light I never knew was there shows up. Eeep! So I call HP. They want me to remove one CPU. Yes, remove one CPU. I tried, and long story short, they dispatch a local guy with a replacement motherboard. Can you send along a FC controller, in case it s dead too? Nope, not until we diagnose a problem with it. Local guy comes out. He s a sharp guy and I really like him. But the motherboard wasn t in stock at the local HP warehouse, so he had to have it driven in from Oklahoma City. He gets here with it by about 4:30. At this point the single most important server to the company s business has been down almost 12 hours. He replaces the motherboard. The server now powers up yay! And it POSTs, and it . doesn t see the disks. !#$!#$ He orders the FC controller, which is so very much not in stock that they can t get it to us until 8:30AM the next morning (keep in mind this thing is on a 4-hour 24/7 contract). Next morning rolls around. Outage now more than 24 hours. He pops the FC controller in, we tweak the SAN settings appropriately, we power up the machine, and . still doesn t see any disks, and the SAN switch still doesn t see any link. EEP! Even the BIOS firmware tool built into the controller doesn t see a link, so we KNOW it s not a software issue. We try plugging and unplugging cables, trying different ports, everything. Nothing makes a difference. At this point, while he ponders what else he can replace while we start migrating the server to a different blade. We get ERP back up on its temporary home an hour later, and he basically orders us every part he can think of while we ve bought him some room. Several additional trips later, he s replaced just about everything at least once, some things 2 or 3 times, and still no FC link. Meanwhile, I ve asked my colleague to submit a new ticket to HP s SAN team so we can try checking of the switch has an issue. They take their sweet time answering until he informs them this morning that it s been *48 HOURS* since we first reported the outage. All of a sudden half a dozen people at HP take a keen interest in our case. As if they could smell this blog post coming So they advise us to upgrade the firmware in the SAN switch, but they also say we really should send this to the blade group; the problem can t be with the SAN and of course the blade people are saying the problem s GOT to be with the SAN . We try to plan the firmware upgrade. In theory, we can lose a switch and nobody ever notices due to multipathing redundancy. In practice, we haven t tested that in 2 years. None of this equipment had even been rebooted in 390 days. While investigating this, we discovered that one of the blade servers could only see one path to its disks, not two. Strange. Fortunately, THAT blade wasn t mission-critical on a Friday, so I power cycled it. And it powered back up. And it promptly lost connection to its disks entirely, causing the SAN switches to display the same mysterious error they did with the first blade the one that nobody at HP had heard of, could find in their documentation, or even on Google. Yes, that s right. Apparently power cycling a server means it loses access to its disks. Faced with the prospect of our network coming to a halt if anything else rebooted (or worse, if the problem started happening without a reboot), we decided we d power cycle one switch now and see what would happen. If it worked out, our problems would be fixed. If not, at least things would go down in our and HP s presence. And that worked? What? Yes. Power cycling the switch fixed every problem over the course of about 2 minutes, without us having to do anything. Meanwhile, HP calls back to say, Uhm, that firmware upgrade we told you to do? DON T DO IT! We power cycle the other switch, and have a normal SAN life again. I let out a WOOHOO! My colleague, however, had the opposite reaction. Now we ll never be able to reproduce this problem to get it fixed! Fair point, I suppose. Then began the fairly quick job of migrating ERP back to its rightful home it s all on Xen already, designed to be nimble for just these circumstances. Full speed restored 4:55PM today. So, to cap it all off, within the space of four hours, we had fail:

• One ERP database
• ERP server s motherboard
• Two fiber optic switches but only regarding their ability to talk to machines recently rebooted
• And possibly one FC controller

Murphy, I hate you. The one fun moment out of this was this conversation: Me to HP guy: So yeah, that machine you ve got open wasn t rebooted in 392 days until today. HP guy: WOW! That s INCRED oh wait, are you running Linux on it? Me: Yep. HP: Figures. No WAY you d get that kind of uptime from Windows. And here he was going to be all impressed.

The module-init-tools upstream maintainers decided that in the future the files in /etc/modprobe.d/ will be processed only if they have a .conf suffix. The latest module-init-tool release complains loudly for each one and still processes them, but this will change.

Please update your packages and remember that the files in this directory are opened and parsed every time modprobe is run (and it is run very often at boot time!), so try to look at the big picture and install a file there only if it is really needed. Do not install a file if it only contains comments, if it is only useful for some unusual environment or if it not needed by the kernel currently in the archive (nowadays most drivers have their own aliases built in, check /lib/modules/$KVER/modules.alias and submit a kernel patch if they are missing).

I start to move some servers when lenny was freeze, and did not have any complain, the update was really smooth; but I still let some servers to be migrated just at the end of the release cycle, and sure, they fail I actually have some fun migrating freeradius and cacti, and note that I actually will have more fun migrating asterisk and openser servers, but for the sake of this release, is the fun that every BOFH should expect, and also, cheers and thanks to everyone who work on this release

That is, my laptop. Since at night I keep it in the same room where I sleep, I figured that I could as well use it as an highly customizable alarm clock. My alarm script uses rtcwake(8) from util-linux-ng to set the RTC alarm clock, suspends the computer and then plays some music at a set volume when it wakes up (if it never wakes up, check your BIOS settings). Since it uses the date(2) command it can parse the alarm time in many formats.

#!/bin/sh -e
# for a random tune, list multiple space-separated files
MUSIC='/home/mp3/The_Velvet_Underground/The_Velvet_Underground_&_Nico/07_Heroin.mp3'
# some days I like different music
DOW="$(date +%u)"
if [ $DOW -eq 6 -o $DOW -eq 7 ]; then
  MUSIC='/home/mp3/The_Velvet_Underground/The_Velvet_Underground_&_Nico/01_Sunday_Morning.mp3'
fi
# choose your favourite music player
player()   mpg123 --random --quiet --control --title "$@"   true;  
#player()   mplayer -loop 0 "$*";  
# play the music at this volume
VOLUME='50%'
# ALSA mixer control used to set and restore the master volume
VOLUMECTL='iface=MIXER,name="Master Playback Volume"'
##############################################################################
# this is more elegant, but it needs rtcwake from util-linux-ng >= 2.14.2
rtcwake_set_alarm()  
  local when="$1"
  local rtc="$2"
  if [ "$rtc" ]; then rtc="--device $rtc"; fi
  local epochtime=$(date --date "$when" +%s)
  [ "$epochtime" ]   return 1
  sudo rtcwake $rtc --mode no --time $epochtime
 
# you can use this function instead if your system lacks a working rtcwake
set_alarm()  
  local when="$1"
  local rtc="$2"
  [ "$rtc" ]   rtc='rtc0'
  local epochtime=$(date --date "$when" +%s)
  [ "$epochtime" ]   return 1
  local alarmfile="/sys/class/rtc/$rtc/wakealarm"
  sudo sh -c "echo 0 > $alarmfile && echo $epochtime > $alarmfile"
 
##############################################################################
if [ "$1" ]; then
  WHEN="$1"
else
  echo "Usage: $0 WHEN"
  exit 1
fi
set_alarm "$WHEN"
# save the volume
oldvolume="$(amixer cget "$VOLUMECTL"   sed -nre '/ : /s/.*=//p')"
amixer -q cset "$VOLUMECTL" $VOLUME
# actually here I run a script which also deals with network interfaces,
# IM and IRC clients and so on
sudo pm-suspend
player $MUSIC
# restore the volume
amixer -q cset "$VOLUMECTL" "$oldvolume"
exit 0

One of the least useful error messages I've ever seen

# dar -N -x backup -K <censor> -g some/where -v
Extracting contents of the archive...
FATAL error, aborting operation
elastic buffer incoherent structure
#

Thanks dar, now it's clear.

These scripts allow automatically running rsnapshot when the right USB hard disk is plugged in. If you do not want so much automation then just let gnome-volume-manager do its work and then run the last script from the command line. /etc/udev/rules.d/68-automount.rules:

ACTION=="add", SUBSYSTEM=="block", \
        ENV ID_FS_USAGE =="filesystem", ENV ID_FS_LABEL =="red", \
        RUN+="backup.agent $env ID_FS_LABEL "

/lib/udev/backup.agent:

#!/bin/sh -e
##############################################################################
wait_for_fs()  
  local fs_path="$1"
  local count="$2"
  if [ -z "$count" ]; then count=60; fi
  while ! mountpoint -q $fs_path; do
    sleep 1
    count=$(($count - 1))
    if [ $count -eq 0 ]; then
      echo "$fs_path has not been mounted after $count seconds!" 1>&2
      return 1
    fi
  done
  return 0
 
##############################################################################
do_stuff()  
  if [ -e /tmp/nobackup ]; then exit 0; fi
  wait_for_fs $FSPATH   exit 1
  nice -n 19 ionice -c 3 sh $FSPATH/doit
  rsnapshot -v diff daily.0 daily.1   mail -s "rsnapshot diff" md
  if [ -e /tmp/noumount ]; then exit 0; fi
  pumount $FSPATH
 
##############################################################################
if [ ! -t 0 ]; then
  exec >> /var/log/backup.log 2>&1
fi
if [ "$1" ]; then
  FSNAME="$1"
else
  echo "Usage: $0 name"
  exit 1
fi
FSPATH="/media/$FSNAME"
# the script may be called multiple times due to multiple change events
# being fired by the kernel
if mountpoint -q $FSPATH; then exit 0; fi
# daemonize
do_stuff &

A while loop which waits for the file system to appear is not elegant, but you cannot do this the right way with only a shell script because it requires watching /proc/mounts using inotify(3). /media/red/doit:

#!/bin/sh -e
##############################################################################
file_older_than_days()  
  local filename="$1"
  local days="$2"
  # consider a day passed if it will end in at most 6 hours
  local more=$((60 * 60 * 6))
  local now=$(date +%s)
  local filetime=$(stat --format=%Y "$filename" 2> /dev/null   echo 0)
  local age=$(($now - $filetime))
  if [ "$age" -gt $((60 * 60 * 24 * $days - $more)) ]; then
    return 0
  else
    return 1
  fi
 
##############################################################################
RSNAPSHOT_ROOT=$(awk '/^snapshot_root[\t ]/   print $2  ' /etc/rsnapshot.conf)
if [ -z "$RSNAPSHOT_ROOT" ]; then
  echo "Cannot determine snapshot_root!"
  exit 1
elif [ ! -d "$RSNAPSHOT_ROOT" ]; then
  echo "snapshot_root is not a directory: $RSNAPSHOT_ROOT"
  exit 1
fi
if file_older_than_days "$RSNAPSHOT_ROOT/daily.0/" 1; then
  if file_older_than_days "$RSNAPSHOT_ROOT/weekly.0/" 14; then
    if file_older_than_days "$RSNAPSHOT_ROOT/monthly.0/" 60; then
      /usr/bin/rsnapshot monthly
    fi
    /usr/bin/rsnapshot weekly
  fi
  /usr/bin/rsnapshot daily
fi